Skip to content

fix(security): migrate to scratch runtime image and update vulnerable dependencies - #79

Open
rkschamer wants to merge 11 commits into
masterfrom
d053727/runtime-image-from-scratch
Open

fix(security): migrate to scratch runtime image and update vulnerable dependencies#79
rkschamer wants to merge 11 commits into
masterfrom
d053727/runtime-image-from-scratch

Conversation

@rkschamer

@rkschamer rkschamer commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Fixes security vulnerabilities by eliminating the attack surface of the runtime image and patching reported CVEs in Go dependencies.

Changes:

  • Scratch runtime image: migrated FROM ubuntu to FROM scratch — no shell, no OS packages, no package-level CVEs in the runtime container
  • In-binary firewall lifecycle: moved NFLOG rule (iptables-nft PREROUTING) and rp_filter sysctl management from the init-container shell scripts into the go-pmtud binary (internal/firewall), which was required to drop the shell dependency from the runtime image
  • Dependency updates (reported vulnerabilities): golang.org/x/net v0.47→v0.58, golang.org/x/sys v0.38→v0.47, k8s.io/{api,apimachinery,client-go} v0.35→v0.37, controller-runtime v0.23→v0.24, prometheus/client_golang v1.23→v1.24

After merge, sapcc/helm-charts#12636 needs to be merged as well (removes init container and preStop hook; both PRs must land together to avoid duplicate NFLOG rules during rollout).

@rkschamer
rkschamer force-pushed the d053727/runtime-image-from-scratch branch 3 times, most recently from 30bc7fb to b1bde83 Compare August 25, 2026 14:56
@rkschamer
rkschamer marked this pull request as draft August 26, 2026 13:32
@rkschamer rkschamer changed the title feat(runtime): enable scratch runtime via in-binary firewall lifecycle [DO NOT MERGE] feat(runtime): enable scratch runtime via in-binary firewall lifecycle Aug 26, 2026
@rkschamer

Copy link
Copy Markdown
Contributor Author

Do not merge this PR, before we verified in QA that this version is working as expected in QA.

We use the image produced by the PR pipeline to deploy this version to QA.

Because of reported vulnerbilities
@rkschamer
rkschamer marked this pull request as ready for review August 31, 2026 07:37
sapcc-bot and others added 8 commits August 31, 2026 10:25
Builds and pushes PR images tagged as ghcr.io/sapcc/go-pmtud:pr-<number>
to allow testing container images before merge.

This file is not managed by go-makefile-maker and will not be overwritten.
* feat(ci): add per-commit SHA tag to PR container image

The PR workflow only produces a mutable pr-<number> tag which gets
overwritten on every push. Keppel mirror caches won't re-pull the
same tag, making it impossible to test updated PR builds.

Add an immutable pr-<number>-<sha> tag alongside the existing one
so each push produces a unique image reference usable for testing.

* fix docker image

---------

Co-authored-by: Rene Kschamer <rene.kschamer@sap.com>
@github-actions

Copy link
Copy Markdown

Merging this branch will increase overall coverage

Impacted Packages Coverage Δ 🤖
github.com/sapcc/go-pmtud/internal/cmd 0.00% (ø)
github.com/sapcc/go-pmtud/internal/firewall 49.30% (+49.30%) 🌟

Coverage by file

Changed files (no unit tests)

Changed File Coverage Δ Total Covered Missed 🤖
github.com/sapcc/go-pmtud/internal/cmd/command.go 0.00% (ø) 0 (-90) 0 0 (-90)
github.com/sapcc/go-pmtud/internal/firewall/manager.go 25.00% (+25.00%) 48 (+48) 12 (+12) 36 (+36) 🌟
github.com/sapcc/go-pmtud/internal/firewall/manager_nonlinux.go 0.00% (ø) 0 0 0
github.com/sapcc/go-pmtud/internal/firewall/rule.go 100.00% (+100.00%) 21 (+21) 21 (+21) 0 🌟
github.com/sapcc/go-pmtud/internal/firewall/sysctl.go 100.00% (+100.00%) 2 (+2) 2 (+2) 0 🌟

Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code.

Changed unit test files

  • github.com/sapcc/go-pmtud/internal/firewall/manager_test.go
  • github.com/sapcc/go-pmtud/internal/firewall/rule_test.go
  • github.com/sapcc/go-pmtud/internal/firewall/sysctl_test.go

@rkschamer

rkschamer commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

Verification: binary rule == script rule

The goal is to confirm that internal/firewall installs the same effective nftables match as the iptables-nft init-container script did.

Prerequisites

A root shell on the target node. Either:

# node-level shell (host filesystem at /host)
kubectl debug node/<node-name> -it  --profile=sysadmin --image=ubuntu -- chroot /host bash

or a privileged pod. The binary's pod is hostNetwork: true / privileged: true, so the rules land directly in the node's nftables.


Step 1 — deploy this PR and capture what the binary installs

After rollout:

sudo nft list table ip pmtud

Expected output:

table ip pmtud {
    chain prerouting {
        type filter hook prerouting priority raw; policy accept;
        iifname "<iface>" meta l4proto icmp icmp type destination-unreachable icmp code frag-needed log group <N>
    }
}

What to compare

Attribute Old (iptables compat) New (binary)
Family ip ip
Hook prerouting prerouting
Priority raw (-300) raw (-300)
iifname default-route iface default-route iface
l4proto icmp icmp
ICMP type destination-unreachable (3) destination-unreachable (3)
ICMP code frag-needed (4) frag-needed (4)
Verdict log group N (non-terminating) log group N (non-terminating)
Table name ip raw (compat shim) ip pmtud (dedicated)

The table name difference is intentional (design decision: clean lifecycle ownership). All match semantics and the hook/priority position are identical.


Step 2 — verify teardown is clean

After the pod is deleted/restarted the table must be gone:

sudo nft list tables | grep pmtud   # should print nothing

Step 3 — Use Metrics to Validate Packet arrival

sum(rate(go_pmtud_recv_packets_total[5m])) — proves nflog group still receives packets ([L92 in pmtud.go](https://github.com/sapcc/go-pmtud/blob/master/internal/nflog/pmtud.go#L92))
sum(rate(go_pmtud_sent_packets_total[5m])) — proves packets are forwarded to each peer ([L145 in pmtud.go](https://github.com/sapcc/go-pmtud/blob/master/internal/nflog/pmtud.go#L145))
rate(go_pmtud_error_total[5m]) — no new errors

No stale rules accumulate across pod restarts — that was the original bug this PR fixes.

@rkschamer rkschamer changed the title [DO NOT MERGE] feat(runtime): enable scratch runtime via in-binary firewall lifecycle [DO NOT MERGE] fix(security): migrate to scratch runtime image and update vulnerable dependencies Aug 31, 2026
@rkschamer rkschamer changed the title [DO NOT MERGE] fix(security): migrate to scratch runtime image and update vulnerable dependencies fix(security): migrate to scratch runtime image and update vulnerable dependencies Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants